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ABSTRACT ' ' ' ' ; 

The author proposes - a greater professional 

^association role in establishing standards for quality assurance in ' 

Itestirig.' He presents his views as a test developer who dislikes the 
legal model for resolving professional issues. The use of - 
publications and informational activities to make people aware of the 
professional standards and how they can be applied is suggested. 
Professional associations -should become actively involved in training 

.and promoting cohtinuing education to familiarize professionals with 
the niw standards and < their application. Compliance with standards 
must / oe monitored through self -policing, tribunals, 
consultation-arbitration, complaint inventory/reporting, or 
facilitators. The National Council on Measurement in Education, 
American Educational Research Association, and the American 
Psychological Association could expand their roles in quality 

.assurance in test development' and use with these, suggested methods. 
(DWH) 
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Perspective of Presentation 

Active test and test program developer 

m t 

Student of criticism of test? and test use 
Prefer to look for mutually agreeable, solutions 
Dislike .intensely legal model for resolving professional Issues/ 
•■ Proponent of greater professional association jrole in quality 
assurance* in testing ' 



) 



[Editorial note: The material placed in boxes in this typed, version was 



included a handout used at' the NCME- meeting. ] 
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tfot*a from presentation at the annual meeting of the National Qouncli on 
Measurement in Education, Chicago, .Illinois, April 1985 

v . * 

Part of invited symposium - Quality Assurance in Test Development and 

Use. The > other papers presented were as follows: . 

Role of the Joint Technical Standards in promoting high quality test 
development and use. - # Rrofessor Robert Linn, University of 
Illinois at Urbana-Champaigri. • , ' ' 1 

Development and operation of a voluntary audit program. - Jerome R. 
Murphy, Educational Testing Service, Princeton, New Jersey. 



/ 



Active teat and test program developer - I began vorkN^ae a test ~* 

developer -almost 25 years ago when 1 served as an item writer for the 

Gates-MacGinitle Reading Tests while I was a graduate student ab 

Columbia University, JTeachers College, After graduate school I 

% i 

worked for 18 year$ in test and test program development at 
Educational Testing Service. Since 1983 I have had the good forKme 
tp bfi Vice Preqid^nt and Elector of the Measurement Division at The 
Psychological Corporation. 

* -> 

•Student of criticisms of teste and test use - I Joined ETI? in J, 965 
just after an intense, period of public criticism of testing. One of 

r 

the major books from' that' period vtas Tyyanny of Testing by Banesh . 

r 

Hoffman. The title alone conveys some of the emotional flavor of the 
attacks. I read all th<| critical articles and books I could find .and 
began a .collection* that now numbers several hundred pieces of suqh 
literature. 



Prefer to look for mutually agreeable solutions - my own style of 

■ft 

dealing with critics of my work is to seek mutually agreeable * 

f solutions, wbat my boss, Thomas A. Williamson, President of Psych 

( Corp, likefa to refer to as "Win-Win 11 outcomes. This, means looking- 

x for what each side wants, building understanding and respect for each 

f 

other, and starting with areas of agreement and helping these areas 
grow larger. I like a counselling or^ negotiating approach as opposed 



to a confrontational* one. 



r. 



•' Dislike intensely 



tojtel for resolving professional issues * I 



have a very negative view of the legal model, for solving problems, y 
It is an adversarial model that focuses' on differences. The goal is 
to win for .your side and the selective presentation of evidence is a 
•critical part* T>f winning. '\ During my titae at ETS I received quite a ' 
bit of coaching on. how to avoid being tricked by other lawyers. One 
'defense is to give as little information as possible.* 

* i » 

Proponent of greater professional association* rolef in quality 

assurance in testing - my presentation reviews the kinds of 

\ 

activities professional associations can and do engage in to promote 
quality test development and use; I urge, a greater role for NCME a« 
well as APA and AERA both because I think It is our responsibility 
- and because I think we can do the job much better with our academic 
and research models than the lawyers will do with their adversarial 
models. 



Setting Professional Standards for ^Test Development and tJse 
Establishing basic standards • 
Using development process to explore, ijiany issues 
Finding areds of professional agreement 
Obtaining broad endorsement of standards 
Publicising ptandards in a-n understandable f orm ^ 

! Keeping standards 'current ^ • 
Providing interpretation when needed ' . , „ 



Establishing basic standards - One of the contributions professional 

—i ■ 1 1 1 ■ 1 ' ' I • - V 

associations can and do make, to quality testing ;Ls tW of deJtting 
standards. This is a time of celebration for us in testing because 
we have a mew and very finely, crafted set of standards for both test 
development and use. . »■ » ^ 

Using develo'pment .process to explore many issues -« The standards\were 

> 

adopted 'after a long process of drafting, deliberation, and review 
that required top measurement professionals to address many important 

\ 

issues of testing policy and .practice. # The many groups involved and.* 

<4ven the conttfbversies over areas of the standards have made it 

i 

\ - • 

likely that a large group of measurement professionals will give 

V . : " ' i 

careful and continuing attention to the staqdrfrds when they plan and 
carry out testing projects, v 

M 

» * 

Finding areas of professional agreement - The process of developing 
standards has also helped clarify areas of agreement and disagreement 
within different parts of the measurement profession* It doesn't :, 
make sensed to set as a general standard something that a large subset 

• ■* 

of the field vietfs as irrelevant, questionable, or evfen absolutely • 

♦ * 

wrong ^ This same process* of having to reach agreement on standards . 

or gaideline? occurred within Educational Testing Service when very % 

« 

different testing programs vie te reviewed against a common set of 
criteria* As Jerry Murphy's paper pointed Qut ETS ran pilot studies 
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applying their guidelines before they became operational, then 

applied the standards internally for^two to three years before 

f 

inviting a visiting committee to perform an external 'audit. This • 
practice of "pretesting" the standards worked very"well and is worth 

... N •. 

Imitating. The review process for the joint standards played this 

• ' * " \ 

sarae pretesting roJV ^ 

Obtaining broad endorsement of standards - The practice of obtaining 
broad endorsement of our professional, standards substantially 
increases their value as action documents'. The ^Psychological 
Corporation -aiyi the other commercial test publishing companies have 
endorsed the standards as have ETS and ACT and a number of *ther < 
-agencies* At The Psychological Corporation, part of the explicit 
responsibility of each measurement professional is to know and be 
guided by the specific standards that bear directly on their* work and 4 ' 
to have a general familiarity with the entire document, 

* jq> ■ 

Publicizing standards in an* understandable,, form - One of the mpjor 

tasks that I see ahead for NCME is that of publicizing the standards 

* 1 
for the, large group of non-technically trained people who. play a 

* ■ * 

major role in testing. The standards document is large and complex 
enough that it will be quite forbidding to someone not trained in*^ 
testing. We need brief summary statements highlighting the s most 
•important issues that are likely to face test users. We also nefid 

4 

i ' * . • 

ways of reaching people in roles such as school system s 
sijperintendencies that require them to make important decisions about 
tests. / 



I— ■ 

Keeping standards current and Providing interpretation when needed - 
We also need some wefr to keep pur standards current. Perhaps those * 
individuals from the drafting committee who have given so much to ' 
"complete the job/ of developing the standards need a recovery period* 
However, we need to establish an ongoing group that can interpret the* 

4 

staqd^rds, issue supplementary advisory statements, and, in general, 
help us conducjt our measurement work in a manner consistent wi\ the 



standards* 



Publications/Informational Activities 

% * ■ ' 
♦ , For professionals within the associations 

• ■* . f 

For others who actually use .tests and test-based d'qfca 
For other interested parties 



For professionals within the associations - One' of the \^yis of making 

* f 

sure that our standards have as significant an effect as possible is 
to use a publications program to make people aware of the standards 
arid how they can be applied ♦ We sqy in the introduction* to the 
November 1, 1984 version - ' ' \ 

► - "The purpose of publishing the standards is to provide 

criteria for the evaluation of tests, testing practices, 

and the effects of test use. 11 ► 

k * ■ • . * 



But whay exactly do we .expect different groups of people to do with 

/ the standards? I know that my test development arid statistical staff 

have to follow development procedures called for in the standards* 

We have to produce test manuals and other related documents so that 
*• * » '» -* ■ « 

the quality of our work can be evaluated. 4 . , 

1 * For others who actually use' thg test and test-based data and For - 

other Interested parties - But what about th$ many other people who*- 
play a. part in testing? What about the people who serve otl test 
selection committees for states and schqpi * districts? We certainly 

f * 

v cannot expt^ct the teachers and .curriculum specialists on thesG- 

» " 4 

conpmittees to read the^ entire standards • However , there is much 
y « valuable- and easy to understand information in the -following chapters: 

3 , Test Development and Revision 

5 Teat Publications 

i v 

. / . • . 

6 General Principles of Test Use u ° 

8 Educational* Testing and Psychological Testing in the 
✓ schools* ■ 

' ' \ ■ . - 

Why not have a series of small publications or handouts ; , with highlights of 

• , y 

these chapters, for use, in test; selection settings? 



One of our very* active NCME people, Ron Hambldton, is inVQlved In a \ 
series of efforts to help with the publications/information effort* 



In hie role as editor of Journal of Educational Measurement , Ron ' 

v t \ [ s ' " ' ' ' '" 

has commissioned a series of seven brief overviews of how the 

( • . . . 

r * standards could be used and barriers to using them. The : 
overviews will be provided by: 

State department of education staff 
School district staff „ ■ ' s - ' 

' Test publishers, 
APA 1 Division //5 Public Affairs Committee - Under Ron's 
chairmanship this group has considered several possible 
activities including the # development of; 
99 Brief guide to using standards 
Series of workshops < 



Training 'Activities 

Initial professional training ^ 
Continuing education for people in field 

Training for those not in field who play major roles, e,g. t school 
/administrators and legislators f , ' 
National anc^ local meetings 



4 Initial professional training - The idea of a series of workshops 
takes tai to another topic ^ that of training. One goal- we should 

set for our professional associations is that of getting useful 

. ( ■ 

material* about the standards into the hands of those who train 
the people we hope to be influencing/ This is not so touch an 
issue for those coming out erf professional measurement programs - 

r 

there are so few such. people, each of us v in this room could 

simply go tell one person and the job would be done* The real 

job is that of reaching- the people^in other professional areas. 
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Continuing education for people in field - Our professional, 
associations do a good job helping us keep up with new \^ 
developments. The journal publication program and the u6e of 
presessions at annual meetings arejftwo effective way? of 
cbmmunicating new knowledge and ideas. to practicing measurement 
professionals. Both these mechanisms can be employed to^increase 

+ 9 ,' 

familiarity with the substance of the Standards and to provide 
help in the application of the standards in different contexts *. 
It might be useful, for example, to have, articles *and workshops 
dealing with tfhe *use of standards In personnel selection, 
classif fcatlon testing in special education, high, school . \ m 
graduatipn/grade promotion testing, teacher certification 
testing, and similar critical arid highly visible, areas of test ■ 
use. ■." , # t 

/ 

t ♦ .. + * 

* •* . • 
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Training for those not in field who play major roles, e«ft, t 

school administrators and legislators - In order for the * 

■■ * 

standards to substahtially improve testing practice we are going 
to haVe to reach the large group of schdol administrators, 
personnel directors, legisfators*, teachers, curriculum 
specialists, school psychologists, and others involved in 

testing.. In some cases we will be able* to work through the state 

■ * . 

departments of 'education, school districts, and state 

r 

professional associations. The professional associations 

concerned with measurement , must review* what is pdssible and 

desirable ai<d then what we can afford. >y Perhaps a self-sustaining 

effort could be developed if we can really meet a need/ A 

Workshop entitled "How to build a legally defensible testing ' 

program" or "20 Steps to staying out of court" would ^very likely ^ 
# 

be well attended. 

National and local meetings - Each Scheduled national, state, or 
local professional meeting over the next two to three years 
should be considered as 1in opportunity to build working knowledge 
of. the standards arid how to apply them to practice. We need riot 
fear redundancy, only the failure to rea^h the people with 
practical, constructive advice and enouragement. .Only such an' 
orchestrated effort Will be sufficient tp* realize the potential ^ 
of the standards to upgrade practice. i 



E. 



Monitoring Compliance With Standards 
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Self-Policing - Judging prof essionals within field*, censure or 
expulsion as ultimate weapon v . ■ / • - 

Tribunal Role ~ Use of hearings on specific instances} or general 
issuep. 

Consultation/Arbitration - Professional association as source oi 
neutral but knowledgeable thtod' parties. 7 

Complatnt Inventory /Reporting - Profe&sidnal association as 
official collecting point fpr concerns - Periodic! reports on t^ie 
nature of concerns. " l , \ K + 

Facilitator - Professional associations as v neutral organizer of 
groups, e.g., possible new consortium including test publishers 
representation. . ■ s a 

• * 
I'Nob My Table" Model - Nofc accepting any responsibility for 
encouraging compliance, '* • 



The area of prbfessional association monitoring of compliance with the 

* j • * • 

standards iseems to t make very nervous the people I have talked to who are 

act lye in AERA and NCME. I hear concerns about expense, time, legal 

4 ■ 

involvements, divisiveness with the field, and the like/ I feel, though, that 

* ■ j 

.leaving compliance to independent individuals and agencies and to the legal 
profession really 'dodges a responsibility. I want to talk, therefore, about a • 
set of professional roles that all seem to jtie to offer some benefits. 

Self-policing - first, self-polioing of association members, this is 
something that APA Has done. To do this you need a cdde of • ethics «f 
some* kind and a method of hearing (Criticism and charges. AERA and 

• / 

NCME are quite different pijganizatiohs than APA yet AERA 'and NCME 
members also make test-related decisions that h*ve significant > 
, impacts on the lives of people. 'The self policing model, .partly 
based on the standards should be considered for both AERA and NCME " 



meofters. 



4 
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Tribunal Rule r Establishment of special .group to hold hearings on 
special techVcal problems associated with such critical arenas ^of 
testing as the following: - , 

Jeacher Testing ' , ■ ¥ 

. ✓» * 

m^rit pay for high test, performance , , -** 

/ J ( * ' ' ' f ^ * # * . * 

^ - standards fpr» in service teacher* "testing 

r - Selecting excellent schools ' 

, > • . • * ' ) 4 

» • * * f * 

Qhooslng students for gifted and talented programs 

Building national Indicators * • ' • 4 

Evaluating computer-based test products* (There is an * . 

* 

APA/AERA/NCM^ semi-formal collaboration on th^s Issue now) 

■ x • •• •• • •• 

^Consultation-Arbit&tion - Provide core of experts as sources of 

knowledge /experience. Could range from Informal referrals based on 

self-nominations or reputation, this alreddy occurs now, to the 
f 

development of a formal group that met certain standards and agreed 

to follow 'particular* guidelines. Perhaps some variation of the EtS 

audit systfcfh cduld be managed by NCME and AERA. People with disputes 

•on technical issues who. had nol; yet gotten* to the stage of wanting to 

sufe each other 1 could .submit "cases 11 for review and judgment. This 

* 

would amount to a "voluntary audit." The result would be a 
professional review and an opinion by people very familiar with the 
standards and the technical issues involved but without a personal 
,\stakevin the dispute. • 



If such a model, were followed I would urge that both parties be 

■ * 

required not to use the results of the $udit in apy future ie^al case. 



Complaint Inventory /Reporting - I have heard APA *ta£f talk about 

occasionally receiving complaints about the use of psychological 

tests, I assume ^hat there are complaints voiced to *the NCME and 

^AERA leadership group also. Perhaps we should, set up -a formal 

mechanism for ehcouraging people with concerns about testing to n 

xfeootd their discontent* The problems cotjid then be referred to the 

agencies jjjho might help and records could be maintained about the 

*■ 

complaints. Annual reporting could be done at our professional 
meetings on the topics/^ssues most frequently raised. Foll6w-up 
checks could be made to see if the concerned party was satisfied with 
his or her treatment. 




t , » 

Facilitator - Our professional organizations can also help improve 

the quality of tests and test uses. by bringing other institutions 

t 

together to pool ideas and talent., "The APA ( is tsrying this now with 

.the test publishers group that Bob Linn mentioned. One aspect of - 

this effort is an attempt to develop a testing industry code of Fair 

Testing — an idea of Gregory Anrig, current President of ETS. 

Another project is that of developing ja set of qualifications for 

test purchasers. % • 

i 

«■ The professional association role is very important in working 
with test publishers as those of us who work for publishers want 
tcbe sure that wt£ observe all ^the laws about relationships .among 

s • competing organisations* 



Our professional associations concerned with testing; also can 

» • 

help by bringing other agencies and associations Into activities 



where we could work together: 



Teachers Groups, 
e.g.* NBA & AFT 

» ■ 

School Adtfiinistrat/rs 



School Psychologists 
Special Educators 
Personnel Directors 



- AASA 

- NASSP 

- NAESP 



( 
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CLOSING 



V 



I have sketched out a number of ways that I think that thiip NCME, AERA, 
and APA could ejxpand their roles ih quality assurance in t^st* development 
and use. Some of my suggestions build on existing activities and expand 
them a bit. Others require a larger professional association commitment 
of people, time, and money. I believe that now is the time to make the 
commitment and to figure out how to get the money that is needed. We are 
experiencing a dramatic increase in* test use and an expansion of; the 
group whose lives are being affected by testing. We must protect our 
Reputations and our professional futures by maintaining and, enhancing 
quality. 
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Lets not follow my last model - th^"Not My Table" one* The label comes 

from my colleague at The Psychological Cprporatlon, Barrle Wellenpl who 

uses It to describe, complete denial of responsibility. In some NYC 

• t 

restaurants, .the waiters are so lmlte*d to their own seciion that even if 
you^ask them the time of day - they say '** • . " » 

"That's not my table" 

Quality in testing on the other hand, is the responsibility of all of us. 
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